In computer vision and computer graphics, 3-dimensional (“3D”) reconstruction is the process of determining the shape and/or appearance of real objects and/or the environment. In general, the term 3D model is used herein to refer to a representation of a 3D environment being modeled by a device. 3D reconstruction may be based on data and/or images of an object obtained from various types of sensors including cameras. For example, a handheld camera may be used to acquire information about a small 3D scene and produce an approximate virtual model of the scene.
Augmented Reality (AR) and Mixed Reality (MR) applications are often used in conjunction with 3D reconstruction. In AR, real images may be processed to add virtual objects to the image. In Mixed Reality (MR) effects, such as Diminished Reality or Mediated Reality, may be implemented based on a 3D model of an existing scene to remove and/or alter objects in the images. In Diminished Reality, a real object is seamlessly removed from the scene, whereas, in Mediated Reality, a real object is seamlessly replaced by a virtual one. Conventional real-time or near real-time AR/MR methods often perform less than optimally because the methods may use inpainting techniques, where portions of the current image are copied over the object to be removed. Inpainting can cause artifacts, which may be exacerbated from a lack of accuracy and/or other errors in scene reconstruction and live camera tracking.
Therefore, there is a need for image processing methods to enhance the quality of rendered MR images to achieve a more optimal user experience.